All committers have signed the CLA.

Hi @anakin87, @Anitha6g and I were trying to think of the best way to handle returning the output of the tool.
A document definitely makes sense for the read and search mode, but what about ground as it returns: Statement, Factuality, Result, Reason.

The unfortunate thing is that it is all markdown and not structured. Should we parse the content and put those things in doc metadata? Thoughts?

Let me have a look ⌛

I'm finally back, after doing a deeper investigation 🕵️ and discussing this with @bilgeyucel.

• We think this component should be named JinaReaderConnector and placed in components/connectors

• While studying the issue with grounding, I found that you can always get a JSON response by all endpoints (reader, search, grounding), if you pass the header 'Accept': 'application/json'. You can play with this and see the results at https://jina.ai/reader/.

• After discovering this JSON response, I would say that we should use this by default because it is more granular and informative (we can add an init parameter called json_response wich defaults to True). This implies that for search we won't have a Document but one for each search result. So I would change the general return type to List[Document] and do something similar to what is discussed below.

  • If json_response==False: return a list of only one Document with only the content, which is a markdown string.
  • If json_response==True, this means we should do some parsing/packing of the results depending on the mode:
    • for reader, create a list of one Document with the returned content as content and all other fields (title, URL, ...) copied into meta
    • for search, create a list of Documents (one for each search result) with the returned content as content and all other fields (title, URL, ...) copied into meta
    • for grounding (not a great solution, but OK), create a list of one Document with the reason field as content and all fields (result, reference, ..., including reason) copied into meta

• By playing with the website, I discovered that there are many other options to run this API which can be set using headers. For this reason, I would add a parameter called headers to the run method, to make these option available to people.

Let me know what you think. I am quite convinced about the plan, even if it requires some work.

@anakin87 @bilgeyucel great points, will get right on it and rebase in a couple days. Thanks for the direction as always 😎

Very nice @jlonge4 , I wanted to use such a component.

@vblagoje thanks very much! Still have to whip up those tests but should be done(ish) today 🤞🏼

@vblagoje @anakin87 what stupid mistake am I making with these test failures and import structure?

@jlonge4 the circular import error should be fixed now.
Integrations mount paths are a bit tricky.

@anakin87 how are we looking?

@jlonge4 in the coming days, I will take a proper look, so as to push this to the finish line.